Abnormality detection apparatus and vehicle system

ABSTRACT

An abnormality detection apparatus includes a feature extraction unit configured to extract an image feature according to a common algorithm, a flow calculation unit, a first abnormality detection unit, and a second abnormality detection unit. An extraction range for the image feature is composed of a predetermined first partial area in a first image, a predetermined second partial area in a second image, and areas near places in the first and second images predicted as destinations of the feature point. The first abnormality detection unit detects an abnormality in the first (second) image based on an optical flow for a feature point in the first (second) partial area. The second abnormality detection unit detects an abnormality by using a feature point in a first overlapped extraction area defined in a first overlapped area and a feature point in a second overlapped extraction area defined in a second overlapped area.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese patent application No. 2017-117746, filed on Jun. 15, 2017, thedisclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

The present disclosure relates to an abnormality detection apparatus anda vehicle system.

An apparatus that detects an abnormality in a camera has been known. Forexample, in techniques disclosed in Japanese Unexamined PatentApplication Publication Nos. 2016-15638 and H11-177964, an abnormalityin a camera is detected by designating or extracting an overlapped areain a plurality of images taken by respective cameras and comparingfeatures in the overlapped area in the images. In such techniques, it isnecessary that shooting ranges of at least two cameras overlap eachother. Therefore, it is impossible to detect an abnormality in a camerawhose shooting range does not overlap with any of the other cameras.

To detect an abnormality in such a camera, there is an abnormalitydetection method using continuity of an optical flow. In this method, afeature point in an image is extracted according to a predeterminedalgorithm (for example, an algorithm disclosed in Jianbo Shi and CarloTomasi, “Good Features to Track”, IEEE Conference on Computer Vision andPattern Recognition, 1994, pages 593-600) and this feature point istracked through temporally consecutive images. Then, by determiningwhether or not the continuity of the movement of this feature point ismaintained, it is possible to detect an abnormality in the camera whoseshooting range does not overlap with any of the other cameras.

SUMMARY

The present inventors have found the following problem. When theshooting ranges of at least two cameras overlap each other, it ispossible to detect an abnormality in these cameras based on features inthe overlapped area in images and also possible to detect an abnormalityin each camera by using the above-described optical flow. However, whenan abnormality is detected by using both of these two methods, aprocessing time for the detection could increase.

Other problems and novel features will become apparent from descriptionsin this specification and the accompanying drawings.

According to one embodiment, an abnormality detection apparatus includesa feature extraction unit configured to extract a feature point and afeature value according to a common extraction algorithm, a flowcalculation unit, a first abnormality detection unit, and a secondabnormality detection unit. Note that an extraction range for thefeature point and the feature value extracted by the feature extractionunit is composed of a predetermined first partial area in a first imagetaken by a first camera, a predetermined second partial area in a secondimage taken by a second camera, and areas near places in the first andsecond images predicted as destinations of the feature point. Further,the first abnormality detection unit detects an abnormality in the firstimage based on an optical flow for the feature point in the firstpartial area and detects an abnormality in the second image based on anoptical flow for the feature point in the second partial area. Further,the second abnormality detection unit detects an abnormality by usingthe feature point in a first overlapped extraction area defined in afirst overlapped area and the feature point in the second overlappedextraction area defined in a second overlapped area.

According to the above-described embodiment, it is possible to, when theshooting ranges of at least two cameras overlap each other, carry outboth of abnormality detection by using an image acquired from one cameraperformed by the first abnormality detection unit and abnormalitydetection by a comparison between at least two images acquired fromrespective cameras performed by the second abnormality detection unitwhile reducing the processing load.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, advantages and features will be moreapparent from the following description of certain embodiments taken inconjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram showing an example of a configuration of anabnormality detection apparatus according to an outline of anembodiment;

FIG. 2 is a schematic diagram showing an example of a positionalrelation among elements constituting a vehicle system according to afirst embodiment;

FIG. 3 is a block diagram showing an example of a hardware configurationof the vehicle system according to the first embodiment;

FIG. 4 is a block diagram showing an example of a functionalconfiguration in the vehicle system according to the first embodiment;

FIG. 5 is a schematic diagram showing a relation among shooting ranges,feature extraction ranges, flow tracking ranges, and overlappedextraction areas;

FIG. 6 is a schematic diagram showing a relation among ranges in imagestaken by cameras;

FIG. 7 is a flowchart showing an example of an operation for anabnormality determination process performed by the vehicle systemaccording to the first embodiment;

FIG. 8 is a schematic diagram showing comparison areas according to asecond embodiment;

FIG. 9 is a block diagram showing an example of a hardware configurationof a vehicle system according to a third embodiment;

FIG. 10 is a block diagram showing an example of a functionalconfiguration in the vehicle system according to the third embodiment;and

FIG. 11 is a flowchart showing an example of an operation for anabnormality determination process performed by the vehicle systemaccording to the third embodiment.

DETAILED DESCRIPTION

For clarifying the explanation, the following descriptions and thedrawings are partially omitted and simplified as appropriate. The samesymbols are assigned to the same elements throughout the drawings andduplicated explanations are omitted as appropriate.

Outline of Embodiment

Prior to describing details of an embodiment, firstly, an outline of theembodiment is given. FIG. 1 is a block diagram showing an example of aconfiguration of an abnormality detection apparatus 1 according to anoutline of an embodiment. The abnormality detection apparatus 1 includesa feature extraction unit 2, a flow calculation unit 3, a firstabnormality detection unit 4, and a second abnormality detection unit 5.

The feature extraction unit 2 receives a first image taken by a firstcamera and a second image taken by a second camera, and extracts imagefeatures (more specifically, feature points and feature values)therefrom. Note that the first and second cameras are cameras that takemoving images. Further, the shooting ranges of the first and secondcameras partially overlap each other. The feature extraction unit 2extracts a feature point and a feature value in the first image and afeature point and a feature value in the second image according to apredetermined extraction algorithm. That is, this predeterminedextraction algorithm is an algorithm that is commonly used when anabnormality is detected by the first abnormality detection unit 4 andwhen an abnormality is detected by the second abnormality detection unit5. In other words, an algorithm for extracting a feature point and afeature value necessary for abnormality detection performed by the firstabnormality detection unit 4 is the same as an algorithm for extractinga feature point and a feature value necessary for abnormality detectionperformed by the second abnormality detection unit 5.

Note that an extraction range for the feature point and the featurevalue extracted by the feature extraction unit 2 is composed of a firstpartial area which is a predetermined partial area in the first image, asecond partial area which is a predetermined partial area in the secondimage, and areas near places in the first and second images predicted asdestinations of the feature point.

The flow calculation unit 3 calculates an optical flow for the featurepoint extracted by the feature extraction unit 2 based on the featurevalue of that feature point. Note that the optical flow is a vectorindicating a movement of a feature point in temporally consecutivetime-series images. That is, the optical flow is a vector indicating amovement of a feature point through consecutive frames of moving imagestaken by a specific camera. For example, when a subject of a camera or acamera itself is moving in a straight line at a constant speed, featurepoints corresponding to the subject move in a fixed direction at aconstant speed through temporally adjacent image frames.

The first abnormality detection unit 4 detects an abnormality in thefirst image based on the optical flow calculated for the first image bythe flow calculation unit 3. Further, the first abnormality detectionunit 4 detects an abnormality in the second image based on the opticalflow calculated for the second image by the flow calculation unit 3.Specifically, the first abnormality detection unit 4 detects anabnormality in the first image based on the optical flow for the featurepoint in the above-described first partial area (i.e., the area in thefirst image where the feature point and the feature value are extracted)and detects an abnormality in the second image based on the optical flowfor the feature point in the second partial area (i.e., the area in thesecond image where the feature point and the feature value areextracted). More specifically, the first abnormality detection unit 4detects an abnormality in the first image based on the optical flow thatis obtained by tracking the feature point located in the above-describedfirst partial area in a predetermined area and detects an abnormality inthe second image based on the optical flow that is obtained by trackingthe feature point located in the second partial area in a predeterminedarea.

Note that an abnormality in an image may be caused by an abnormality (afailure) in a camera that has been used to take that image, or may becaused by an abnormality in one of processes that are performed fromwhen image information is output from the camera to when this imageinformation is input to the feature extraction unit 2 (i.e., a failurein an apparatus that perform this process). As described above, theabnormality detection performed by the first abnormality detection unit4 is performed by using an image acquired from one camera to detect anabnormality in this camera or an abnormality in a specific process thathas been performed for image information output from this camera.

The second abnormality detection unit 5 detects an abnormality in thefirst or second image based on a result of a comparison between a firstindex value for a feature point in a first overlapped area in the firstimage and a second index value for a feature point in a secondoverlapped area in the second image. Note that the first overlapped areais an overlapped area in the first image in which the shooting ranges inthe first and second images overlap each other. The second overlappedarea is an overlapped area in the second image in which the shootingranges in the first and second images overlap each other. Therefore, thesame subject is present in the first and second overlapped areas.

More specifically, the second abnormality detection unit 5 detects anabnormality by using a feature point in a first overlapped extractionarea defined in the first overlapped area and a feature point in asecond overlapped extraction area defined in the second overlapped area.Note that the first overlapped extraction area is an area in the firstpartial area that corresponds to the second partial area. Note that thefirst overlapped extraction area can also be regarded as being an areain the first partial area that overlaps the second partial area.Further, the second overlapped extraction area is an area in the secondpartial area that corresponds to the first partial area. Note that thesecond overlapped extraction area can also be regarded as being an areain the second partial area that overlaps the first partial area. Thatis, a subject that is present in an overlapped extraction area in animage is an object for which features are extracted in both the firstand second images.

Note that similarly to the first abnormality detection unit 4, anabnormality in an image detected by the second abnormality detectionunit 5 may be caused by an abnormality (a failure) in a camera that hasbeen used to take that image, or may be caused by an abnormality in oneof processes that are performed from when image information is outputfrom the camera to when this image information is input to the featureextraction unit 2 (i.e., a failure in an apparatus that perform thisprocess).

Further, as an example, the above-described index value is an opticalflow for a feature point that starts from that feature point in thefirst overlapped area. However, the index value may be a feature valueof the feature point in the first overlapped area. Note that in FIG. 1,broken-line arrows extending to the second abnormality detection unit 5indicate such optionality of the index value. That is, when theabove-described index value is an optical flow, the path that extendsfrom the flow calculation unit 3 to the second abnormality detectionunit 5 may exist, but the path that extends from the feature extractionunit 2 to the second abnormality detection unit 5 may not exist.Further, when the index value is a feature value, the path that extendsfrom the flow calculation unit 3 to the second abnormality detectionunit 5 may not exist, but the path that extends from the featureextraction unit 2 to the second abnormality detection unit 5 may exist.

It should be noted that in FIG. 1, the arrows connecting elements aremerely examples of relations among the elements and are shown for easierunderstanding. That is, they are not shown to exclude relations amongelements other than those shown in the abnormality detection apparatus 1shown in FIG. 1.

As described above, in the abnormality detection apparatus 1, the firstabnormality detection unit 4 performs abnormality detection using imagesacquired from one camera and the second abnormality detection unit 5performs abnormality detection by comparing at least two images acquiredfrom respective cameras. That is, since the abnormality detectionapparatus 1 has two types of abnormality detection functions, i.e.,abnormality detection by the first abnormality detection unit 4 andabnormality detection by the second abnormality detection unit 5,accuracy of the abnormality detection is improved compared to the casewhere only one type of abnormality detection is performed.

Further, in the abnormality detection apparatus 1, image featuresnecessary for abnormality detection performed by the first abnormalitydetection unit 4 and those necessary for abnormality detection performedby the second abnormality detection unit 5 are not necessarily extractedby using different extraction algorithms. Therefore, an extractionresult obtained by the feature extraction unit 2 can be used for bothabnormality detection performed by the first abnormality detection unit4 and abnormality detection performed by the second abnormalitydetection unit 5. That is, by using a common extraction algorithm, it ispossible to carry out the image feature extraction process withoutperforming complicated processes. Further, in the abnormality detectionapparatus 1, a partial area is used as an extraction range for imagefeatures, instead of using the entire image area. Therefore, it ispossible to reduce the load necessary for the image feature extractionprocess.

As described above, the abnormality detection apparatus 1 makes itpossible to, when the shooting ranges of at least two cameras overlapeach other, carry out both abnormality detection by the firstabnormality detection unit 4 and abnormality detection by the secondabnormality detection unit 5 while reducing the processing load.

First Embodiment

Next, details of an embodiment are described. FIG. 2 is a schematicdiagram showing an example of a positional relation among elementsconstituting a vehicle system 10 according to a first embodiment. Thevehicle system 10 includes cameras 101A to 101D as information inputdevices installed in a vehicle 100 and other components installed in thevehicle 100. Note that in the example shown in FIG. 2, four cameras areshown. However, the vehicle system 10 needs to be equipped with only twocameras or more. In the example shown in FIG. 2, control devices such asa brake 102 and a steering wheel 103, and an information output devicesuch as a warning display device 104 are shown as other components.Further, the vehicle system 10 includes an ECU (Electronic Control Unit)105.

The cameras 101A to 101D, the brake 102, the steering wheel 103, and thewarning display unit 104 are connected to the ECU 105 that recognizesinformation, determines a vehicle state, controls the vehicle, andperforms communication. The ECU 105 may be a single component or may becomposed of a plurality of components for respective functions. When theECU 105 is composed of a plurality of components, these components areconnected so that information can be mutually exchanged among them.

Note that the cameras 101A to 101D are cameras that take moving imagesof the surroundings of the vehicle 100 (i.e., a scene around the vehicle100). The camera 101A is a camera installed on the front side of thevehicle 100 so that its main shooting range is in front of the vehicle100. The camera 101B is a camera installed on the right side of thevehicle 100 so that its main shooting range is on the right side of thevehicle 100. The camera 101C is a camera installed on the rear side ofthe vehicle 100 so that its main shooting range is behind the vehicle100. The camera 101D is a camera installed on the left side of thevehicle 100 so that its main shooting range is on the left side of thevehicle 100.

Note that the camera 101A is installed in the vehicle 100 so that itsshooting range partially overlaps the shooting range of the camera 101Band partially overlaps the shooting range of the camera 101D. The camera101B is installed in the vehicle 100 so that its shooting rangepartially overlaps the shooting range of the camera 101A and partiallyoverlaps the shooting range of the camera 101C. The camera 101C isinstalled in the vehicle 100 so that its shooting range partiallyoverlaps the shooting range of the camera 101B and partially overlapsthe shooting range of the camera 101D. The camera 101D is installed inthe vehicle 100 so that its shooting range partially overlaps theshooting range of the camera 101C and partially overlaps the shootingrange of the camera 101A. It is assumed that the overlapped ranges ofthe cameras do not change with time.

Next, the configuration of the vehicle system 10 is described in a moredetailed manner with reference to FIG. 3. FIG. 3 is a block diagramshowing an example of a hardware configuration of the vehicle system 10.As shown in FIG. 3, the ECU 105 includes a recognition MCU (MicroController Unit) 110, a determination MCU 111, and control MCUs 112 and113. Note that when the vehicle system 10 supports connection of anextension camera 106 that is added to take moving images, the ECU 105further includes an extension-camera MCU 114. Note that more than oneextension camera 106 may be added. It should be noted that the addedextension camera 106 may be installed in the vehicle 100 so that itsshooting range overlaps the shooting range of at least one of otherextension cameras 106 or at least one of the cameras 101A to 101D.

The recognition MCU 110 is an MCU for recognizing images taken by thecameras 101A to 101D and images taken by the extension camera 106. Therecognition MCU 110 is connected to the cameras 101A to 101D throughbuses 120A to 120D, respectively.

The determination MCU 111 is an MCU for determining a state of thevehicle 100. The determination MCU 111 is connected to the warningdisplay device 104 through a bus 121 and controls contents displayed inthe warning display device 104.

The control MCU 112 is a MCU for controlling the vehicle 100 inaccordance with an input from the steering wheel 103 and is connected tothe steering wheel 103 through a bus 122. The control MCU 113 is an MCUfor controlling the vehicle 100 in accordance with an input from thebrake 102 and is connected to the brake 102 through a bus 123. Thecontrol MCUs 112 and 113 do not necessarily have to be separate MCUs.That is, they may be formed as one MCU.

The extension-camera MCU 114 is a MCU for processing signals input fromthe extension camera 106. The extension-camera MCU 114 performs, forexample, processing similar to those performed by camera I/Fs 142A to142D and a capture unit 143 (which are described later) and generatesimage data. Further, the extension-camera MCU 114 compresses image datain order to reduce an amount of data that is output to an intra-vehiclenetwork 125. Note that the image data output from the extension-cameraMCU 114 is stored in an external memory 130 through the intra-vehiclenetwork 125 and the recognition MCU 110. When the extension-camera MCU114 is provided, the extension camera 106 is connected to theextension-camera MCU 114 through a bus 124.

The recognition MCU 110, the determination MCU 111, the control MCUs 112and 113, and the extension-camera MCU 114 are connected to theintra-vehicle network 125.

Further, the ECU 105 includes an external memory 130. The externalmemory 130 is a memory in which images taken by the cameras 101A to 101Dand images taken by the extension camera 106 are stored.

Details of a configuration of the recognition MCU 110 are describedhereinafter. The recognition MCU 110 includes an intra-vehicle networkI/F 140, an external memory I/F 141, camera I/Fs 142A to 142D, a captureunit 143, an image processing processor 150, an image recognitionprocessor 160, and a CPU (Central Processing Unit) 170.

The intra-vehicle network I/F 140 is an interface forinputting/outputting information from/to the intra-vehicle network 125.The external memory I/F 141 is an interface for inputting/outputtinginformation from/to the external memory 130. The camera I/Fs 142A to142D are interfaces for converting signals captured in the cameras 101Ato 101D into signals in a signal format that can be accepted by thecapture unit 143. For example, the camera I/Fs 142A to 142D areinterfaces for converting input serial signals into parallel signals andoutputting them to the capture unit 143.

Note that these interfaces may be connected to a dedicated network, ormay be connected to a general-purpose network such as a CAN (ControllerArea Network) or an Ethernet. Further, it is also possible to implementa connection to a general-purpose network by switching the same bus in atime-division manner or the like.

The capture unit 143 is a circuit that generates image data based onsignals output from the cameras 101A to 101D. The image data output fromthe capture unit 143 is stored in the external memory 130 through theexternal memory I/F 141.

The image processing processor 150 is a circuit that performspredetermined image processing for images taken by the cameras 101A to101D and the extension camera 106. The image processing processor 150reads image data stored in the external memory 130 through the externalmemory I/F 141, performs predetermined image processing, and then storesthe resultant image data, i.e., the data that has been subjected to theimage processing in the external memory 130 through the external memoryI/F 141. Note that examples of the predetermined image processingperformed by the image processing processor 150 include a process fordecoding compressed image data and a correction for a distortion intaken image data. However, the predetermined image processing is notlimited these examples.

The image recognition processor 160 is a circuit that performs an imagerecognition process. The image recognition processor 160 performs animage recognition process for images that have been processed by theimage processing processor 150. The image recognition processor 160reads image data stored in the external memory 130 through the externalmemory I/F 141, performs a predetermined image recognition process, andthen stores a result of the image recognition process in the externalmemory 130 through the external memory I/F 141. Specifically, as theimage recognition process, the image recognition processor 160 extractsimage features (more specifically, a feature point and a feature value)and calculates an optical flow therefor.

The CPU 170 carries out various processes by executing a program storedin the external memory 130 or other memories. The CPU 170 performs anabnormality detection process for detecting an abnormality in an imagebased on the image recognition result obtained by the image recognitionprocessor 160.

In this embodiment, the image recognition processor 160 and the CPU 170correspond to the abnormality detection apparatus 1 shown in FIG. 1.That is, the image recognition processor 160 and the CPU 170 perform anabnormality detection process based on image data which has beensubjected to the image processing performed by the image processingprocessor 150. For example, when a failure occurs in one of the cameras101A to 101D, the camera I/Fs 142A to 142D, the capture unit 143, andthe image processing processor 150, an abnormality occurs in an imagethat is input to the image recognition processor 160. In thisembodiment, a failure in the cameras 101A to 101D, the camera I/Fs 142Ato 142D, the capture unit 143, and the image processing processor 150 isdetected based on an abnormality in an image which occurs when thefailure occurs in these circuits. Note that when the extension camera106 is used, the extension camera 106 and the extension camera MCU 114are also included in the target circuits in which a failure is detected.

Next, a functional configuration of the vehicle system 10 related to theabnormality detection process is described. FIG. 4 is a block diagramshowing an example of a functional configuration related to theabnormality detection process performed in the vehicle system 10. Asshown in FIG. 4, the vehicle system 10 includes a feature extractionunit 200, a flow calculation unit 201, a first abnormality detectionunit 202, a second abnormality detection unit 203, and a comprehensivedetermination unit 204 as the configuration of the abnormality detectionapparatus. Note that the feature extraction unit 200 and the flowcalculation unit 201 are implemented by the image recognition processor160. Further, the first abnormality detection unit 202, the secondabnormality detection unit 203, and the comprehensive determination unit204 are implemented by the CPU 170.

The feature extraction unit 200 corresponds to the feature extractionunit 2 shown in FIG. 1. That is, the feature extraction unit 200extracts image features (i.e., a feature point and a feature value) ofan image taken by each camera (i.e., each of the cameras 101A to 101Dand the extension camera 106) according to a predetermined extractionalgorithm. Note that since the cameras 101A to 101D and the extensioncamera 106 take moving images, the feature extracting unit 200 extractsimage features for each of temporally consecutive image frames. In thisembodiment, the feature extraction unit 200 extracts image features inimages that have been subjected to image processing performed by theimage processing processor 150.

Examples of the aforementioned predetermined extraction algorithminclude known algorithms disclosed in scientific papers, such as Harrisand GoodFeaturesToTrack. Regarding the feature points, for example,points by which an outline of a subject can be acquired in images andwhose movements through frames can be traced are extracted. Such featurepoints exist, for example, in corners, but are not limited to corners.Further, the feature extraction unit 200 calculates feature values forextracted feature points according to the aforementioned predeterminedextraction algorithm.

The feature extraction unit 200 performs an extraction process for anarea in a feature extraction range that is defined in advance for eachcamera (see feature extraction ranges R1 _(A) and R1 _(B) in FIGS. 5 and6) and for areas near places estimated as destinations of the featurepoint. Note that a relation between the feature extraction range andother ranges will be described later with reference to FIGS. 5 and 6.The positions in the image estimated as the destinations of the featurepoint are estimated by, for example, the flow calculation unit 201.

The flow calculation unit 201 corresponds to the flow calculation unit 3shown in FIG. 1. The flow calculation unit 201 calculates an opticalflow for the feature point extracted by the feature extraction unit 200based on the feature value of the feature point. Specifically, when afeature point in an (n−1)-th image frame I_(n−1) is expressed as afeature point A_(n−1), the flow calculation unit 201 calculates (i.e.,determines) which direction a corresponding feature point A_(n)extracted in an n-th image frame I_(n), which is the frame next to theimage frame I_(n−1), has moved in by comparing the feature values of thefeature points. In this way, the flow calculation unit 201 calculates avector indicating the movement of the feature point through a pluralityof frames. For example, the flow calculation unit 201 calculates theoptical flow by using a known algorithm disclosed in scientific papers(such as OpticalFlowLK). Note that the flow calculation unit 201calculates (i.e., determines) an optical flow starting from the featurepoint extracted by the feature extraction unit 200 in a flow trackingrange defined in advance for each camera (see flow tracking ranges R2_(A) and R2 _(B) in FIGS. 5 and 6). Note that a relation between theflow tracking range and other ranges will be described later.

Further, the flow calculation unit 201 estimates a place at adestination of the feature point. The flow calculation unit 201estimates a place of a feature point A_(n+1) in a (n+1)-th image frameI_(n+1) from the optical flow that has been already calculated based onimage frames up to the n-th image frame. For example, the flowcalculation unit 201 estimates the place of the feature point A_(n+1) inthe (n+1)-th image frame I_(n+1) under the assumption that the subjector the camera is moving substantially in a straight line at a constantspeed. Then, the feature extraction unit 200 performs a process forextracting an image feature for an area near the place estimated by theflow calculation unit 201 (e.g., for an image area within apredetermined distance from the estimated place).

The first abnormality detection unit 202 corresponds to the firstabnormality detection unit 4 shown in FIG. 1. That is, the firstabnormality detection unit 202 detects an abnormality in an imageacquired from the camera 101A based on an optical flow calculated forthe image from the camera 101A by the flow calculation unit 201.Further, the first abnormality detection unit 202 detects an abnormalityin an image acquired from the camera 101B based on an optical flowcalculated for the image from the camera 101B by the flow calculationunit 201. Further, the first abnormality detection unit 202 detects anabnormality in an image acquired from the camera 101C based on anoptical flow calculated for the image from the camera 101C by the flowcalculation unit 201. Further, the first abnormality detection unit 202detects an abnormality in an image acquired from the camera 101D basedon an optical flow calculated for the image from the camera 101D by theflow calculation unit 201. Further, when the extension camera 106 isprovided, the first abnormality detection unit 202 detects anabnormality in an image acquired from the extension camera 106 based onan optical flow for the image from the extension camera 106. In thisway, the first abnormality detection unit 202 detects an abnormality inan image acquired from one camera by using the image from this camera.

Abnormality detection performed by the first abnormality detection unit202 is further explained. Note that although abnormality detection foran image acquired from the camera 101A is explained hereinafter as anexample, abnormality detection for images acquired from other camerasare performed in a similar manner. The first abnormality detection unit202 detects an abnormality in an image acquired from the camera 101Abased on ab optical flow starting from a feature point in a featureextraction range R1. The first abnormality detection unit 202 detects anabnormality in an image by determining continuity of an optical flow onthe presumption that moving images are acquired by the camera when asubject or the camera is moving substantially in a straight line at aconstant speed.

A determination of continuity of an optical flow made by the firstabnormality detection unit 202 is described hereinafter. When temporalchanges (i.e., changes over time) of the moving direction and the movingamount of the optical flow in the image acquired from the camera arewithin predetermined ranges, the first abnormality detection unit 202determines that the image from the camera is normal. In contrast tothis, when an optical flow in the image acquired from the camera ofwhich temporal changes of the moving direction and the moving amount arewithin the predetermined ranges cannot be obtained, such as whentemporal changes of the moving direction and the moving amount of theoptical flow are not within the predetermined ranges, the firstabnormality detection unit 202 determines that the image from the camerais abnormal.

When the subject of the camera 101A or the camera 101A itself is movingsubstantially in a straight line at a constant speed, the movingdirection and the moving amount of a feature point through temporallyadjacent frames that are acquired from the camera 101A at apredetermined frame rate become constant. Therefore, the firstabnormality detection unit 202 determines whether or not a differencebetween a moving direction P_(n) and a moving direction P_(n+1) iswithin a permissible range P_(R) and determines whether or not a movingdistance Q_(n) and a moving distance Q_(n+1) is within a permissiblerange Q_(R). Note that the moving direction P_(n) is a moving directionfrom a feature point A_(n−1) in an (n−1)-th image frame I_(n−1) to itscorresponding feature point A_(n) in an n-th image frame I_(n). Themoving direction P_(n+1) is a moving direction from the feature pointA_(n) to its corresponding feature point A_(n+1) in an image frameI_(n+1). Further, the moving distance Q_(n) is a moving distance fromthe feature point A_(n−1) to the feature point A_(n), and the movementdistance Q_(n+1) is a moving distance from the feature point A_(n) tothe feature point A_(n+1).

The first abnormality detection unit 202 determines that the image isnormal when the difference between the moving directions P_(n) andP_(n+1) is within the permissible range P_(R) and the difference betweenthe moving distances Q_(n) and Q_(n+1) is within the permissible rangeQ_(R). Further, the first abnormality detection unit 202 determines thatthe image is abnormal in all the other cases. For example, when somefailure occurs in any of the circuits located between the camera 101Aand the image recognition processor 160 or in the camera 101A itself andhence images are not supplied from the camera 101A at regular intervals,the optical flow of which the temporal changes of the moving directionand the moving amount are within the predetermined ranges cannot beobtained. Therefore, the first abnormality detection unit 202 detects anabnormality in the image. Note that in the abnormality detection by thefirst abnormality detection unit 202, the subject or the camera does notnecessarily have to be moving absolutely in a straight line at aconstant speed. That is, the only requirement for the abnormalitydetection by the first abnormality detection unit 202 is that thesubject or the camera should be moving. In other words, the movingdirection and the moving speed of the subject or the camera may bechanged. This is because the time between adjacent image frames issufficiently small and the permissible range is defined for thedetermination as described above.

The second abnormality detection unit 203 corresponds to the secondabnormality detection unit 5 shown in FIG. 1. That is, the secondabnormality detection unit 203 detects an abnormality in an image usingan optical flow for a feature point in an overlapped area of a pluralityof images taken by respective cameras. The second abnormality detectionunit 203 requires at least two cameras whose shooting ranges partiallyoverlap each other in order to detect an abnormality in an image.Details of the second abnormality detection unit 203 are explainedhereinafter by using a camera 101A and a camera 101B whose shootingranges partially overlap each other. However, the second abnormalitydetection unit 203 performs a similar process for other combinations ofcameras whose shooting ranges overlap each other.

The second abnormality detection unit 203 detects an abnormality in animage acquired from the camera 101A or 101B based on a result of acomparison between an optical flow for a feature point in an overlappedextraction area in an image from the camera 101A (see an overlappedextraction area R3 _(A) in FIGS. 5 and 6) and an optical flow for afeature point in an overlapped extraction area in an image from thecamera 101B (see an overlapped extraction area R3 _(B) in FIGS. 5 and6). Note that the overlapped extraction area in the image acquired fromthe camera 101A corresponds to the above-described first overlappedextraction area, and is an area in the feature extraction range in theimage from the camera 101A (see the feature extraction range R1 _(A) inFIGS. 5 and 6) that corresponds to the feature extraction range in theimage acquired from the camera 101B (see the feature extraction range R1_(B) in FIGS. 5 and 6). In other words, the overlapped extraction areain the image acquired from the camera 101B corresponds to theabove-described second overlapped extraction area, and is an area in thefeature extraction range in the image from the camera 101B (see thefeature extraction range R1 _(B) in FIGS. 5 and 6) that corresponds tothe feature extraction range in the image acquired from the camera 101A(see the feature extraction range R1 _(A) in FIGS. 5 and 6).

A relation among the shooting range, the feature extraction range, theflow tracking range, and the overlapped extraction area is explainedhereinafter with reference to FIGS. 5 and 6. FIG. 5 is a schematicdiagram showing a relation among shooting ranges, feature extractionranges, flow tracking ranges, and overlapped extraction areas. Note thatFIG. 5 shows the relation when viewed from above the vehicle 100. FIG. 6is a schematic diagram showing a relation among ranges in an image takenby the camera 101A (a left part in FIG. 6) and a relation among rangesin an image taken by the camera 101B (a right part in FIG. 6). Note thatthe images shown in FIG. 6 are rectangular images that are obtained byperforming predetermined image processing such as a decoding process anda correction for a distortion for images acquired from the camera byusing the image processing processor 150. The feature extraction unit200 and the flow calculation unit 201 process such rectangular images.Although FIGS. 5 and 6 show ranges related only to the cameras 101A and101B, similarly ranges are also defined for the other cameras. Arelation among the ranges is explained hereinafter with reference toFIGS. 5 and 6.

In FIG. 5, a shooting range R_(A) of the camera 101A and a shootingrange R_(B) of the camera 101B are indicated by ranges enclosed withthick solid lines. In FIG. 6, each of the shooting ranges R_(A) andR_(B) is an area indicated by the outermost rectangular frame. In FIG.5, the shooting ranges R_(A) and R_(B) partially overlap each other. Thefeature extraction range R1 _(A) is defined in advance as a part of thearea of the shooting range R_(A). Similarly, the feature extractionrange R1 _(B) is defined in advance as a part of the area of theshooting range R_(B). In particular, the feature extraction range R1_(A) partially extends into and overlap the shooting range R_(B) of theother camera 101B. Similarly, the feature extraction range R1 _(B)partially extends into and overlap the shooting range R_(A) of the othercamera 101A. In other words, at least a part of the feature extractionrange is defined in at least a part of the area that overlaps theshooting range of the other camera. Note that in FIGS. 5 and 6, thefeature extraction ranges (R1 _(A) and R1 _(B)) are indicated by finehatching. The feature extraction range R1 _(A) is also referred to as afirst partial area and the feature extraction range R1 _(B) is alsoreferred to as a second partial area.

For the camera 101A, feature extraction ranges R1 _(A) are defined onboth sides of the shooting range R_(A). One of the feature extractionrange R1 _(A) is a feature extraction range that is defined so as toinclude a part of the area overlapping the shooting range R_(B) of thecamera 101B, and the other feature extraction range R1 _(A) is a featureextraction range that is defined so as to include a part of the areaoverlapping the shooting range of the camera 101D (not shown).Similarly, for the camera 101B, feature extraction ranges R1 _(B) aredefined on both sides of the shooting range R_(B). One of the featureextraction range R1 _(B) is a feature extraction range that is definedso as to include a part of the area overlapping the shooting range R_(A)of the camera 101A, and the other feature extraction range R1 _(B) is afeature extraction range that is defined so as to include a part of thearea overlapping the shooting range of the camera 101C (not shown). Notethat in the example shown in FIG. 5, the feature extraction range isdefined in the part of the area where the shooting ranges of bothcameras overlap each other. However, the feature extraction range may bedefined over the entire area where the shooting ranges of both camerasoverlap each other.

In FIGS. 5 and 6, flow tracking ranges are indicated by areas filledwith dots (R2 _(A) and R2 _(B)). The flow tracking range is defined soas to include the feature extraction ranges therein. In other words, theflow tracking range is defined in a wider area than the featureextraction range. Specifically, the flow tracking range R2 _(A) for thecamera 101A is defined so as to include both of the feature extractionranges R1 _(A) located on both sides. Similarly, the flow tracking rangeR2 _(B) for the camera 101B is defined so as to include both of thefeature extraction ranges R1 _(B) located on both sides. Therefore, thefeature extraction range R1 _(A) is also (a part of) the flow trackingrange R2 _(A). Further, the feature extraction range R1 _(B) is also (apart of) the flow tracking range R2 _(B). Note that in FIGS. 5 and 6,the flow tracking range is defined in a part of the area of the shootingrange of the camera. However, the flow tracking range may be definedover the entire shooting range of the camera.

In FIGS. 5 and 6, the overlapped extraction areas (R3 _(A) and R3 _(B))are areas indicated by coarse hatching. The overlapped extraction areais a part where the feature extraction range in an image taken by one ofthe cameras and the feature extraction range in an image taken by theother camera overlap each other in the shooting range. The overlappedextraction area R3 _(A) for the camera 101A is also (a part of) thefeature extraction range R1 _(A) and (a part of) the flow tracking rangeR2 _(A). Similarly, the overlapped extraction area R3 _(B) for thecamera 101B is also (a part of) the feature extraction range R1 _(B) and(a part of) the flow tracking range R2 _(B).

The second abnormality detection unit 203 detects an abnormality in animage by using an optical flow that is obtained by using a predeterminedarea in an overlapped area in an image taken by the camera 101A thatoverlaps an image taken by the camera 101B as a tracking range for thatoptical flow and an optical flow that is obtained by using apredetermined area in an overlapped area in the image taken by thecamera 101B that overlaps the image taken by the camera 101A as atracking range for that optical flow. That is, for the abnormalitydetection process performed by the second abnormality detection unit203, a feature point in the image of the camera 101A is tracked withinthe predetermined area in the overlapped area of the image of the camera101A. Similarly, for the abnormality detection process performed by thesecond abnormality detection unit 203, a feature point in the image ofthe camera 101B is tracked within the predetermined area in theoverlapped area of the image of the camera 101B. In the followingdescription, this predetermined area in the overlapped area is alsoreferred to as a comparison area. Note that the shooting range of thecomparison area in the one of the images corresponds to the shootingrange of the comparison area in the other image. That is, a subject thatis present in the comparison area of one of the images is also presentin the comparison area of the other image. In this embodiment, thecomparison area is rectangular areas R4 _(A) and R4 _(B) defined in theoverlapped extraction areas R3 _(A) and R3 _(B) (see FIG. 6). However,the comparison area may be the overlapped extraction areas R3 _(A) andR3 _(B). Note that details of the rectangular areas R4 _(A) and R4 _(B)will be described later.

In contrast to this, the first abnormality detection unit 202 detects anabnormality in images taken by the camera 101A and 101B by using anoptical flow that is obtained by using a range wider than the trackingrange used in the second abnormality detection unit 203 as its trackingrange. Specifically, for the abnormality detection process performed bythe first abnormality detection unit 202, a feature point in the imageof the camera 101A is tracked in the flow tracking range R2 _(A).Similarly, for the abnormality detection process performed by the firstabnormality detection unit 202, a feature point in the image of thecamera 101B is tracked in the flow tracking range R2 _(B).

Details of the abnormality detection performed by the second abnormalitydetection unit 203 are further explained. The second abnormalitydetection unit 203 detects an abnormality by comparing a first frequencydistribution, which indicates the number of samples (a frequency) foreach direction for an optical flow for a feature point in a comparisonarea in an image acquired from the camera 101A, and a second frequencydistribution, which indicates the number of samples (a frequency) foreach direction for an optical flow for a feature point in a comparisonarea in an image acquired from the camera 101B. Note that in thisembodiment, the comparison area is the rectangular areas R4 _(A) and R4_(B) as described above. However, the comparison area may be theoverlapped extraction areas R3 _(A) and R3 _(B).

For example, as the aforementioned first frequency distribution, thesecond abnormality detection unit 203 creates a histogram of movingdirections for an optical flow of a feature point in the comparisonarea. Further, as the aforementioned second frequency distribution, thesecond abnormality detection unit 203 creates a histogram of movingdirections for an optical flow of a feature point in the comparisonarea. Then, the second abnormality detection unit 203 calculates anormalized cross-correlation between elements of both histograms anddetermines whether or not both histograms are similar to each other.When the second abnormality detection unit 203 determines that bothhistograms are similar to each other, i.e., the first and secondfrequency distributions are similar to each other, it determines thatboth of the images acquired from the cameras 101A and 101B are normal.When the second abnormality detection unit 203 determines that bothhistograms are not similar to each other, i.e., the first and secondfrequency distributions are not similar to each other, it determinesthat one of the images acquired from the cameras 101A and 101B isabnormal. Note that since the second abnormality detection unit 203detects an abnormality by performing a comparison, it cannot determinewhich of the images of the cameras 101A and 101B is abnormal.

Note that as described above, in this embodiment, the second abnormalitydetection unit 203 creates a frequency distribution by using an opticalflow for a feature point in a pre-designated rectangular area in theoverlapped extraction area. That is, in FIG. 6, the determinationprocess is performed by using a feature point in the rectangular area R4_(A), which is a predetermined comparison area, indicated by a thicksolid-line frame in the overlapped extraction area R3 _(A) and a featurepoint in the rectangular area R4 _(B), which is a predeterminedcomparison area, indicated by a thick solid-line frame in the overlappedextraction area R3 _(B). As described above, in this embodiment, in theprocess performed by the second abnormality detection unit 203, thefeature point in the predetermined rectangular area included in theoverlaid extraction area is used. However, a feature point in the wholeoverlapped extraction area or a feature point in an area having apredetermined arbitrary shape in the overlapped extraction area may beused. Note that the area having the predetermined shape in theoverlapped extraction area is defined in advance in, for example, thedesigning phase of the vehicle system 10 so that it does not extend tothe outside of the overlapped extraction area.

It should be noted that when the shooting direction of the camera 101Adiffers from that of the camera 101B as in the case of this embodiment,the moving direction of a subject in an image of the camera 101A differsfrom the moving direction of that subject in an image of the camera101B. That is, when a given subject has spatially moved in a directionX, while a feature point of the subject in an image acquired from thecamera 101A moves in, for example, a direction X_(A), a feature point ofthat subject in an image acquired from the camera 101B moves in, forexample, a direction X_(B). Therefore, when the shooting directions ofthe cameras 101A and 101B differ from each other, the second abnormalitydetection unit 203 creates a histogram for an optical flow in which themoving direction is normalized. Therefore, a rotation process isperformed for the optical flow so that the direction X_(A) is normalizedto the direction X, and a rotation process is performed for the opticalflow so that the direction X_(B) is normalized to the direction X. Arotation angle in the rotation process is different for each camera andaccording to the shooting direction of each camera. For example, therotation angle is defined in advance in the designing phase of thevehicle system 10. Therefore, when the shooting directions of thecameras 101A and 101B differ from each other, the second abnormalitydetection unit 203, for example, rotates the moving direction calculatedby the flow calculation unit 201 for the feature point in the image ofthe camera 101A by a predefined first angle and then creates a frequencydistribution. Similarly, the second abnormality detection unit 203rotates the moving direction calculated by the flow calculation unit 201for the feature point in the image of the camera 101B by a predefinedsecond angle and then creates a frequency distribution. However, whenthe shooting directions of the cameras 101A and 101B are the same aseach other, the above-described rotation process is unnecessary.

Next, the comprehensive determination unit 204 is described. Thecomprehensive determination unit 204 makes a final determination on thepresence/absence of an abnormality in an image based on detectionresults of the first and second abnormality detection units 202 and 203.Therefore, the determinations on the presence/absence of an abnormalitymade by the first and second abnormality detection units 202 and 203 canbe considered to be provisional (or temporary) determinations that aremade prior to the final determination.

The comprehensive determination unit 204 may make a determination basedon detection results of the first and second abnormality detection units202 and 203 in accordance with a predetermined final determination rule.Any arbitrary rule can be used as the final determination rule. However,as an example, the comprehensive determination unit 204 makes a finaldetermination in accordance with the following rule.

In this embodiment, the comprehensive determination unit 204 makes afinal determination while giving a higher priority to the detectionresult of the second abnormality detection unit 203 than to thedetection result of the first abnormality detection unit 202. That is,when the detection result of the first abnormality detection unit 202for an image of a given camera does not match the detection result ofthe second abnormality detection unit 203, the comprehensivedetermination unit 204 makes a final determination on whether or not theimage of this camera is abnormal by adopting the detection result of thesecond abnormality detection unit 203. This is because while thedetection by the first abnormality detection unit 202 is an abnormalitydetermination by using images taken by only one camera, the detection bythe second abnormality detection unit 203 is performed by comparingimages taken by at least two cameras. Therefore, in view of the factthat a possibility that similar abnormalities occur in both of theimages to be compared is small, the determination accuracy for thedetection performed by the second abnormality detection unit 203 ishigher than that of the first abnormality detection unit 202.

Further, in this embodiment, the comprehensive determination unit 204handles the detection performed by the second abnormality detection unit203 as follows. Note that the following descriptions are given by usingimages taken by the cameras 101A and 101B as images to be compared inthe second abnormality detection unit 203. However, the comprehensivedetermination unit 204 performs a similar process for each of the othercombinations of cameras whose shooting ranges overlap each other.

The comprehensive determination unit 204 determines whether or not thedetection result of the second abnormality detection unit 203 for theimages of the cameras 101A and 101B should be used for the finaldetermination based on the number of samples in a frequency distribution(a histogram) (i.e., the total number of optical flows for eachdirection) for the image of the camera 101A and the number of samples ina frequency distribution (a histogram) (i.e., the total number ofoptical flows for each direction) for the image of the camera 101B.

It is considered that when the number of samples in the frequencydistribution is less than a predetermined number, the reliability of theresult of the comparison using this frequency distribution is low.Therefore, for example, when the number of samples in the frequencydistribution of the image of the camera 101A and the number of samplesin the frequency distribution of the image of the camera 101B are bothless than a predetermined number, the comprehensive determination unit204 does not use the detection result of the second abnormalitydetection unit 203 obtained based on these frequency distributions forthe final determination. Note that when the numbers of samples in thefrequency distributions are both less than the predetermined number, thecomparison process performed by the second abnormality detection unit203 may be skipped.

On the other hand, for example, when the number of samples in thefrequency distribution of the image of the camera 101A and the number ofsamples in the frequency distribution of the image of the camera 101Bare both equal to or greater than the predetermined number, thecomprehensive determination unit 204 uses the detection result of thesecond abnormality detection unit 203 obtained based on these frequencydistributions for the final determination. That is, in this case, thecomprehensive determination unit 204 makes the final determination whiletaking account of the detection result of the second abnormalitydetection unit 203 based on these frequency distributions.

Note that when the number of samples in the frequency distribution ofone of the images to be compared is less than the predetermined numberand the number of samples in the frequency distribution of the other ofthe images to be compared is equal to or greater than the predeterminednumber, the comprehensive determination unit 204 handles the detectionresult of the second abnormality detection unit 203 obtained based onthese frequency distributions as follows. In this case, when the secondabnormality detection unit 203 determines that the images of the cameras101A and 101B are normal, the comprehensive determination unit 204 usesthe detection result of the second abnormality detection unit 203obtained based on these frequency distributions for the finaldetermination. On the other hand, when the second abnormality detectionunit 203 determines that the image of the camera 101A or 101B isabnormal, the comprehensive determination unit 204 does no use thedetection result of the second abnormality detection unit 203 obtainedbased on these frequency distributions for the final determination.

Note that even when the detection result obtained by the comparisonbetween the frequency distributions of the images of the cameras 101Aand 101B is not used in the comprehensive determination unit 204, adetection result obtained by a comparison between a frequencydistribution of an image of the camera 101A and a frequency distributionof an image of the camera 101D, which is located adjacent to the camera101A, could be used in the comprehensive determination unit 204. This isbecause there are cases in which while the total number of optical flowsfor a feature point in one of the comparison areas of the camera 101A issmall, the total number of optical flows for a feature point in theother comparison area is larger. Similarly, a detection result obtainedby a comparison between a frequency distribution of an image of thecamera 101B and a frequency distribution of an image of the camera 101C,which is located adjacent to the camera 101B, could be used in thecomprehensive determination unit 204.

Next, an operation for the abnormality determination process performedby the vehicle system 10 according to this embodiment is described. FIG.7 is a flowchart showing an example of an operation for the abnormalitydetermination process performed by the vehicle system 10 according tothe first embodiment. The operation for the abnormality determinationprocess performed by the vehicle system 10 is described hereinafter withreference to the flowchart shown in FIG. 7.

In a step 100 (S100), images of moving images taken by the cameras 101Ato 101D are input to the image recognition processor 160. Note that whenthe extension camera 106 is provided, images of moving images taken bythe extension camera 106 are also input to the image recognitionprocessor 160.

Next, in a step 101 (S101), the feature extraction unit 200 extractsimage features of the image taken by each camera.

Next, in a step 102 (S101), the flow calculation unit 201 calculatesoptical flows for the feature points extracted by the feature extractionunit 200.

Next, processes in steps 103 and 104 (S103 and S104) are performed. Notethat the processes in the steps 103 and 104 (S103 and S104) may beperformed in parallel with each other as shown in FIG. 7, or may beperformed one after another.

In the step 103, the first abnormality detection unit 202 performs anabnormality detection process by using the optical flow obtained in thestep 102. Further, in the step 104, the second abnormality detectionunit 203 performs an abnormality detection process by using the opticalflow obtained in the step 102. That is, the result of the processes inthe steps 101 and 102 is shared (i.e., used) by both of the first andsecond abnormality detection unit 202 and 203.

Next, in a step 105 (S105), the comprehensive determination unit 204makes a final determination on the presence/absence of an abnormality inan image based on the detection results of the first and secondabnormality detection units 202 and 203. When the image is determined tobe abnormal by the comprehensive determination unit 204 (Yes in step 106(S106)), i.e., when it is determined that a failure has occurred, apredetermined fail-safe process is carried out (step 107 (S107)). Forexample, when the comprehensive determination unit 204 finallydetermines that the image is abnormal, it notifies the determination MCU111 of the occurrence of the abnormality. Upon receiving thenotification about the occurrence of the abnormality, the determinationMCU 111 performs a fail-safe process such as displaying a warning in thewarning display device 104 and initializing the recognition MCU 110.When the image is determined to be normal by the comprehensivedetermination unit 204 (No in step 106 (S106)), the normal operation iscarried out (i.e., continued) (step 108 (S108)).

The first embodiment has been described above. In the vehicle system 10,the first abnormality detection unit 202 performs abnormality detectionusing images acquired from one camera and the second abnormalitydetection unit 203 performs abnormality detection by comparing at leasttwo images acquired from respective cameras. Therefore, accuracy of theabnormality detection can be improved compared to the case where onlyone type of abnormality detection is performed. Further, both of theimage features for the first abnormality detection unit 202 and thosefor the second abnormality detection unit 203 are extracted by using thecommon extraction algorithm. Therefore, it is possible to efficientlycarry out the process for extracting image features. Further, in thevehicle system 10, a partial area in the image, rather than the entirearea of the image, is used as an extraction range for the image featurefor a starting point of an optical flow. Therefore, it is possible toreduce the load for the process for extracting image features.Therefore, the vehicle system 10 makes it possible to carry out both ofabnormality detection by the first abnormality detection unit 202 andabnormality detection by the second abnormality detection unit 203 whilereducing the processing load.

Further, the first abnormality detection unit 202 detects an abnormalityin an image based on temporal changes of an optical flow obtained fromimages acquired from one camera. Therefore, it is possible to carry outabnormality detection by using images acquired from one camera.

Further, the second abnormality detection unit 203 detects anabnormality in an image by using an optical flow for a feature point inan overlapped area of a plurality of images acquired from respectivecameras. Therefore, since both of the first and second abnormalitydetection unit 202 and 203 detect an abnormality in an image by using anoptical flow, the result of the process for calculating the optical flowcan be shared (i.e., used) by both of the first and second abnormalitydetection units 202 and 203. That is, it is possible to reduce the loadfor the process related to the abnormality detection.

Further, the first abnormality detection unit 202 detects an abnormalityin an image by using an optical flow that is obtained by using a rangewider than the tracking range used in the second abnormality detectionunit 203 as its tracking range. The detection by the second abnormalitydetection unit 203 is performed by comparing overlapped areas of aplurality of images. Therefore, the area that can be used for thedetection is limited. In contrast to this, in the detection by the firstabnormality detection unit 202, it is possible to perform theabnormality detection by using a wider area in addition to theoverlapped area. It is desirable that the tracking area be large inorder to reliably detect an optical flow. In this embodiment, asdescribed above, the first abnormality detection unit 202 uses a rangewider than the tracking range used in the second abnormality detectionunit 203 as its tracking range. Therefore, it is possible to detect anoptical flow for the first abnormality detection unit 202 more reliablycompared to the case where the first abnormality detection unit 202 usesthe same range as the tracking range used in the second abnormalitydetection unit 203 as its tracking range. In other words, it is possibleto improve accuracy of the abnormality detection performed by the firstabnormality detection unit 202 compared to the case where the trackingranges of the first and second abnormality detection units 202 and 203are same as each other.

Further, the second abnormality detection unit 203 detects anabnormality by comparing a frequency distribution for an optical flow inone of the images with a frequency distribution of an optical flow inthe other image. Therefore, it is possible to compare images by usingoptical flows.

Further, the comprehensive determination unit 204 determines whether ornot the result of the abnormality detection by the second abnormalitydetection unit 203 should be used for the final determination based onthe number of samples in the frequency distributions. Therefore, it ispossible to prevent, when the number of samples in the frequencydistribution is small and hence the reliability of the frequencydistribution is low, the result of the abnormality detection by thesecond abnormality detection unit 203 from being reflected in (i.e.,used for) the final determination.

Further, the comprehensive determination unit 204 makes a finaldetermination while giving a higher priority to the detection result ofthe second abnormality detection unit 203 than to the detection resultof the first abnormality detection unit 202. Therefore, a detectionresult by the second abnormality detection unit 203, in which images ofat least two cameras are compared and hence whose reliability is higherthan that in the determination technique using one camera, ispreferentially used. Therefore, it is possible to make a more reliablefinal determination.

Second Embodiment

Next, a second embodiment is described. In the first embodiment, thecomparison area has a rectangular shape. That is, the second abnormalitydetection unit 203 creates a frequency distribution by using an opticalflow for a feature point in a pre-designated rectangular area in anoverlapped extraction area (e.g., the rectangular area R4 _(A) or R4_(B)). In contrast to this, a comparison area has a circular shape inthe second embodiment. That is, in the second embodiment, a frequencydistribution is creased by using an optical flow for a feature point ina pre-designated circular area (e.g., a rectangular area R5 _(A) or R5_(B) in FIG. 8) in an overlapped extraction area. The second embodimentis described hereinafter. However, descriptions of configurations andoperations similar to those in the first embodiment are omitted.

As described above, when the shooting directions of a plurality ofcameras to be compared differ from each other, it is necessary to createa histogram of optical flows in which moving directions are normalized.That is, it is necessary to perform a process for rotating by apredetermined angle for each camera and then create a histogram.Therefore, in this embodiment, a comparison area for images taken bycameras is specified by its center coordinates (x, y), a radius r, and arotational angle θ. Note that the rotation angle θ is an angle fornormalizing the moving direction and is an angle that is determined inadvance for each camera based on its shooting direction.

FIG. 8 is a schematic diagram showing a comparison area according to thesecond embodiment. As shown in FIG. 8, the second abnormality detectionunit 203 according to the second embodiment uses, among optical flowsfor feature points calculated for images taken by the camera 101A by theflow calculation unit 201, optical flows for feature points in acircular area R5 _(A), which is obtained by rotating a range determinedby center coordinates (x_(A), y_(A)) and a radius r_(A) by an angleθ_(A), as optical flows to be compared, and by doing so, creates afrequency distribution in which the moving directions of the opticalflows are rotated by the angle θ_(A). Similarly, the second abnormalitydetection unit 203 according to the second embodiment uses, amongoptical flows for feature points calculated for images taken by thecamera 101B by the flow calculation unit 201, optical flows for featurepoints in a circular area R5 _(B), which is obtained by rotating a rangedetermined by center coordinates (x_(B), y_(B)) and a radius r_(B) by anangle θ_(B), as optical flows to be compared, and by doing so, creates afrequency distribution in which the moving directions of the opticalflows are rotated by the angle θ_(B). Then, the second abnormalitydetection unit 203 detects an abnormality in an image by comparing thesefrequency distributions. Note that in this embodiment, since thecomparison area has a circular shape, the range of the comparison areadoes not change even when the comparison area is rotated by the angleθ_(A) or θ_(B).

In the first embodiment, the comparison area has a rectangular shape. Inthis case, assume that, for example, a comparison area for an imagetaken by a camera is specified by coordinates (x, y) of its upper-leftcorner, a width w of the rectangular, a height h of the rectangular, anda rotation angle θ. Alternatively, assume that the comparison area isspecified by coordinates (x0, y0) of its upper-left corner, coordinates(x1, y1) of the lower-right corner, and a rotation angle θ. In thiscase, the area that is obtained by rotating the rectangular areaspecified by the coordinates by the rotation angle θ becomes a rangethat differs from the rectangular range specified by the coordinates.Therefore, the range that is defined based on the coordinates does notcoincide with the range that is actually compared. As a result, there isa possibility that the comparison area of the camera 101A does notcorresponds to the comparison area of the camera 101B. In contrast tothis, in this embodiment, the comparison area has a circular shape andits range does not change even when it is rotated. Therefore, it ispossible to make the comparison area of the camera 101A correspond tothe comparison area of the camera 101B and thereby to detect anabnormality more accurately.

Third Embodiment

Next, a third embodiment is described. In the first and secondembodiments, an abnormality is determined based solely on informationobtained from images. In contrast to this, in this embodiment,information about a movement of the vehicle is taken into account forthe determination of an abnormality in an image acquired from a camera.The third embodiment is described hereinafter. However, descriptions ofconfigurations and operations similar to those in the first and secondembodiments are omitted.

FIG. 9 is a block diagram showing an example of a hardware configurationof a vehicle system 11 according to the third embodiment. The vehiclesystem 11 differs from the vehicle system 10 shown in FIG. 3 in that thevehicle system 11 further includes a GNSS device 107 and a positioningMCU 115.

The GNSS device 107 is a terminal that acquires positioning informationof the vehicle 100 based on a GNSS (Global Navigation Satellite System)signal. The positioning MCU 115 is provided in the ECU 105. Thepositioning MCU 115 is connected to the GNSS device 107 through a bus126 and calculates direction information indicating a moving directionof the vehicle 100 and speed information indicating a moving speed ofthe vehicle 100 based on the positioning information acquired by theGNSS device 107. Further, the positioning MCU 115 is connected to theintra-vehicle network 125 and outputs a calculation result to therecognition MCU 110.

FIG. 10 is a block diagram showing an example of a functionalconfiguration related to an abnormality detection process performed inthe vehicle system 11. As shown in FIG. 10, the vehicle system 11includes a feature extraction unit 200, a flow calculation unit 201, aninformation acquisition unit 205, a first abnormality detection unit202, a second abnormality detection unit 203, a third abnormalitydetection unit 206, and an comprehensive determination unit 204 as theconfiguration of the abnormality detection apparatus. That is, theconfiguration of the vehicle system 11 differs from that shown in FIG. 4in that the information acquisition unit 205 and the third abnormalitydetection unit 206 are added in the vehicle system 11. Similarly to thefirst and second abnormality detection units 202 and 203, and thecomprehensive determination unit 204, the information acquisition unit205 and the third abnormality detection unit 206 are implemented by theCPU 170.

The information acquisition unit 205 acquires direction informationindicating a moving direction of the vehicle 100 and speed informationindicating a moving speed of the vehicle 100. Specifically, theinformation acquisition unit 205 acquires direction information andspeed information calculated by the positioning MCU 115. However, theinformation source is not limited to the above-described example. Thatis, the information acquisition unit 205 may acquire the directioninformation and the speed information of the vehicle from any otherdevices. For example, the information acquisition unit 205 may acquiredirection information of the vehicle 100 that is calculated by thecontrol MCU 112 based on a steering angle measured by the steering wheel103, and may acquire speed information output from a vehicle-speedsensor (not shown).

When the moving speed and the moving direction of the vehicle, and theshooting direction of a camera are obtained, a movement vector of afeature point included in an image taken by this camera can becalculated. Therefore, the third abnormality detection unit 206estimates an optical flow for the image of this camera based oninformation acquired by the information acquisition unit 205 and apredetermined shooting direction of the camera. Then, the thirdabnormality detection unit 206 detects an abnormality in an image ofthis camera by comparing the estimated optical flow with an actualoptical flow for the image of the camera calculated by the flowcalculation unit 201. The third abnormality detection unit 206 performsa process for detecting an abnormality in an image for each camera.

The third abnormality detection unit 206 detects an abnormality in animage by comparing a frequency distribution (a histogram) indicating thenumber of samples (a frequency) for each direction for an optical flowfor a feature point extracted from a feature extraction range and afrequency distribution (a histogram) indicating the number of samples (afrequency) for each direction for an optical flow estimated frominformation acquired from the information acquisition unit 205.Specifically, the third abnormality detection unit 206 calculates anormalized cross-correlation between elements of both frequencydistributions and determines whether or not both frequency distributionsare similar to each other. When both frequency distributions are similarto each other, the third abnormality detection unit 206 determines thatthe image to be compared is normal. When both frequency distributionsare not similar to each other, the third abnormality detection unit 206determines that the image to be compared is abnormal. For example, whena feature point included in an image taken by a given camera is notmoving over time even though the vehicle 100 is moving, the thirdabnormality detection unit 206 can detect that an abnormality hasoccurred in the image taken by this camera.

Note that in the abnormality detection by the second abnormalitydetection unit 203, it is impossible to specify which of the images tobe compared the detected abnormality has occurred in. In contrast tothis, in the abnormality detection by the third abnormality detectionunit 206, it is possible to specify which of the images of the camerasthe abnormality has occurred in as in the case of the first abnormalitydetection unit 202.

The comprehensive determination unit 204 according to this embodimentmakes a final determination on the presence/absence of an abnormality inan image based on detection results of the first, second and thirdabnormality detection units 202, 203 and 206. Therefore, thedeterminations on the presence/absence of an abnormality made by thefirst, second and third abnormality detection units 202, 203 and 206 canbe considered to be provisional (i.e., temporary) determinations thatare made prior to the final determination.

The comprehensive determination unit 204 may make a determination basedon detection results of the first, second and third abnormalitydetection units 202, 203 and 206 in accordance with a predeterminedfinal determination rule. Any arbitrary rule can be used as the finaldetermination rule.

As described above, regarding the determination result by the secondabnormality detection unit 203, when the number of samples in thefrequency distribution is less than a predetermined number, a reliabledetection result cannot be obtained from the second abnormalitydetection unit 203. Further, in the determination by the firstabnormality detection unit 202, a correct determination cannot beobtained unless the feature point in the image is moving. Therefore, forexample, the comprehensive determination unit 204 may use adetermination made by the third abnormality detection unit 206 as afinal determination when a reliable determination result cannot beobtained from the second abnormality detection unit 203 and adetermination result also cannot be obtained from the first abnormalitydetection unit 202.

Next, an operation for the abnormality determination process performedby the vehicle system 11 according to this embodiment is described. FIG.11 is a flowchart showing an example of an operation for the abnormalitydetermination process performed by the vehicle system 11 according tothe third embodiment. The flowchart shown in FIG. 11 is obtained byadding a step 150 (S150) and a step 151 (S151) in the flowchart shown inFIG. 7. Further, in the flowchart shown in FIG. 11, the step 105 (S105)shown in the flowchart of FIG. 7 is replaced by a step 152 (S152). Theother steps shown in FIG. 11 are similar to those in FIG. 7. Differencesfrom the flowchart shown in FIG. 7 are described hereinafter, in whichduplicated explanations are omitted.

In the operation for the abnormality determination process performed bythe vehicle system 11, the information acquisition unit 205 acquiresspeed information indicating a moving speed of the vehicle 100 anddirection information indicating a moving direction of the vehicle 100in a step 150 (S150). Note that in the example shown in FIG. 11, thestep 150 is performed after the step 102. However, the step 150 may beperformed at any timing before the process performed by the thirdabnormality detection unit 206 in the step 151.

After the step 150, the third abnormality detection unit 206 detects anabnormality by estimating an optical flow from the information obtainedin the step 150 and comparing the estimated optical flow with theoptical flow calculated in the step 102 in a step 151 (S151). Note thatthe third abnormality detection unit 206 performs the abnormalitydetection process for each camera.

Note that the processes in the steps 103, 104 and 150 may be performedin parallel with each other as shown in FIG. 11, or may be performed oneafter another.

Next, in a step 152 (S152), the comprehensive determination unit 204makes a final determination on the presence/absence of an abnormality inan image based on the detection results of the first, second and thirdabnormality detection units 202, 203 and 206.

The third embodiment has been described above. The vehicle system 11according to this embodiment includes the third abnormality detectionunit 206 in addition to the first and second abnormality detection unit202 and 203. Therefore, it is possible to detect an abnormality in animage more reliably.

The present disclosure made by the inventors of the present applicationhas been explained above in a concrete manner based on embodiments.However, the present disclosure is not limited to the above-describedembodiments, and needless to say, various modifications can be madewithout departing from the spirit and scope of the present disclosure.For example, each component shown in FIGS. 1, 4 and 10 may be partiallyor entirely implemented by a hardware circuit, or may be implemented bysoftware. When the component is implemented by software, it can beimplemented by, for example, having a processor such as a CPU execute aprogram loaded in a memory.

Further, the program can be stored and provided to a computer using anytype of non-transitory computer readable media. Non-transitory computerreadable media include any type of tangible storage media. Examples ofnon-transitory computer readable media include magnetic storage media(such as floppy disks, magnetic tapes, hard disk drives, etc.), opticalmagnetic storage media (e.g. magneto-optical disks), CD-ROM (compactdisc read only memory), CD-R (compact disc recordable), CD-R/W (compactdisc rewritable), and semiconductor memories (such as mask ROM, PROM(programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random accessmemory), etc.). The program may be provided to a computer using any typeof transitory computer readable media. Examples of transitory computerreadable media include electric signals, optical signals, andelectromagnetic waves. Transitory computer readable media can providethe program to a computer via a wired communication line (e.g. electricwires, and optical fibers) or a wireless communication line.

The first to third embodiments can be combined as desirable by one ofordinary skill in the art.

While the disclosure has been described in terms of several embodiments,those skilled in the art will recognize that the disclosure can bepracticed with various modifications within the spirit and scope of theappended claims and the disclosure is not limited to the examplesdescribed above.

Further, the scope of the claims is not limited by the embodimentsdescribed above.

Furthermore, it is noted that, Applicant's intent is to encompassequivalents of all claim elements, even if amended later duringprosecution.

What is claimed is:
 1. An abnormality detection apparatus comprising: afeature extraction circuit configured to extract a feature point and afeature value in a first image taken by a first camera, and a featurepoint and a feature value in a second image taken by a second cameraaccording to a predetermined extraction algorithm, a shooting range ofthe second camera partially overlapping a shooting range of the firstcamera; a flow calculation circuit configured to calculate an opticalflow for the feature point of the first or second images extracted bythe feature extraction circuit based on the feature value of thatfeature point of the first or second images, respectively; a firstabnormality detection circuit configured to detect an abnormality in thefirst image based on the optical flow for the first image calculated bythe flow calculation circuit and detect an abnormality in the secondimage based on the optical flow for the second image calculated by theflow calculation circuit; and a second abnormality detection circuitconfigured to detect an abnormality in the first or second image basedon a result of a comparison between a first index value for the featurepoint of the first image in a first overlapped area and a second indexvalue for the feature point of the second image in a second overlappedarea, the first overlapped area being an area in the first image thatoverlaps the second image, the second overlapped area being an area inthe second image that overlaps the first image, wherein an extractionrange for the feature point and the feature value of the first andsecond images extracted by the feature extraction circuit is composed ofa first partial area which is a predetermined partial area in the firstimage, a second partial area which is a predetermined partial area inthe second image, and areas near places in the first and second imagespredicted as destinations of the feature point of the first and secondimages, the first abnormality detection circuit detects an abnormalityin the first image based on an optical flow for the feature point in thefirst partial area and detects an abnormality in the second image basedon an optical flow for the feature point in the second partial area, thesecond abnormality detection circuit detects an abnormality by using thefeature point in a first overlapped extraction area defined in the firstoverlapped area and the feature point in a second overlapped extractionarea defined in the second overlapped area; the first overlappedextraction area is an area in the first partial area that corresponds tothe second partial area, and the second overlapped extraction area is anarea in the second partial area that corresponds to the first partialarea, and the predetermined extraction algorithm is commonly used whenany abnormality is detected by the first abnormality detection circuitand when any abnormality is detected by the second abnormality detectioncircuit.
 2. The abnormality detection apparatus according to claim 1,wherein the first index value is an optical flow for the feature pointin the first overlapped extraction area, and the second index value isan optical flow for the feature point in the second overlappedextraction area.
 3. The abnormality detection apparatus according toclaim 2, wherein the second abnormality detection circuit compares anoptical flow for the feature point in a predetermined circular areadesignated in the first overlapped extraction area with an optical flowfor the feature point in a predetermined circular area designated in thesecond overlapped extraction area.
 4. The abnormality detectionapparatus according to claim 2, wherein the second abnormality detectioncircuit uses an optical flow that is obtained by using a predeterminedarea in the first overlapped area as a tracking range for that opticalflow, and an optical flow that is obtained by using a predetermined areain the second overlapped area as a tracking range for that optical flow,and the first abnormality detection circuit detects an abnormality inthe first and second images by using an optical flow that is obtained byusing a range wider than the tracking range used in the secondabnormality detection circuit as its tracking range.
 5. The abnormalitydetection apparatus according to claim 2, wherein the second abnormalitydetection circuit detects an abnormality by comparing a first frequencydistribution and a second frequency distribution, the first frequencydistribution being a frequency distribution for numbers of samples foreach direction for the optical flow for the feature point in the firstoverlapped extraction area, the second frequency distribution being afrequency distribution for numbers of samples for each direction for theoptical flow for the feature point in the second overlapped extractionarea.
 6. The abnormality detection apparatus according to claim 5,further comprising a comprehensive determination circuit configured tomake a final determination on presence/absence of an abnormality in thefirst or second image based on detection results of the first and secondabnormality detection circuits, wherein the comprehensive determinationcircuit determines whether or not a detection result of the secondabnormality detection circuit should be used for the final determinationbased on the number of samples in the first frequency distribution andthe number of samples in the second frequency distribution.
 7. Theabnormality detection apparatus according to claim 1, further comprisinga comprehensive determination circuit configured to make a finaldetermination on presence/absence of an abnormality in the first orsecond image based on detection results of the first and secondabnormality detection circuits, wherein the comprehensive determinationcircuit makes the final determination while giving a higher priority tothe detection result of the second abnormality detection circuit than tothe detection result of the first abnormality detection circuit.
 8. Theabnormality detection apparatus according to claim 1, wherein the firstabnormality detection circuit detects an abnormality in the first imagebased on whether or not temporal changes of a moving direction and amoving amount of the optical flow for the first image are withinpredetermined ranges, and detects an abnormality in the second imagebased on whether or not temporal changes of a moving direction and amoving amount of the optical flow for the second image are withinpredetermined ranges.
 9. A vehicle system comprising: a first cameradisposed in a vehicle, the first camera being configured to shoot asurrounding of the vehicle; a second camera disposed in the vehicle sothat its shooting range partially overlaps a shooting range of the firstcamera, the second camera being configured to shoot a surrounding of thevehicle; a feature extraction circuit configured to extract a featurepoint and a feature value in a first image taken by the first camera,and a feature point and a feature value in a second image taken by thesecond camera according to a predetermined extraction algorithm; a flowcalculation circuit configured to calculate an optical flow for thefeature point of the first or second images extracted by the featureextraction circuit based on the feature value of that feature point ofthe first or second images, respectively; a first abnormality detectioncircuit configured to detect an abnormality in the first image based onthe optical flow for the first image calculated by the flow calculationcircuit and detect an abnormality in the second image based on theoptical flow for the second image calculated by the flow calculationcircuit; and a second abnormality detection circuit configured to detectan abnormality in the first or second image based on a result of acomparison between a first index value for the feature point of thefirst image in a first overlapped area and a second index value for thefeature point of the second image in a second overlapped area, the firstoverlapped area being an area in the first image that overlaps thesecond image, the second overlapped area being an area in the secondimage that overlaps the first image, wherein an extraction range for thefeature point and the feature value of the first and second imagesextracted by the feature extraction circuit is composed of a firstpartial area which is a predetermined partial area in the first image, asecond partial area which is a predetermined partial area in the secondimage, and areas near places in the first and second images predicted asdestinations of the feature point of the first and second images, thefirst abnormality detection circuit detects an abnormality in the firstimage based on an optical flow for the feature point in the firstpartial area and detects an abnormality in the second image based on anoptical flow for the feature point in the second partial area, thesecond abnormality detection circuit detects an abnormality by using thefeature point in a first overlapped extraction area defined in the firstoverlapped area and the feature point in a second overlapped extractionarea defined in the second overlapped area; the first overlappedextraction area is an area in the first partial area that corresponds tothe second partial area, and the second overlapped extraction area is anarea in the second partial area that corresponds to the first partialarea, and the predetermined extraction algorithm is commonly used whenany abnormality is detected by the first abnormality detection circuitand when any abnormality is detected by the second abnormality detectioncircuit.
 10. The vehicle system according to claim 9, furthercomprising: an information acquisition circuit configured to acquiredirection information indicating a moving direction of the vehicle andspeed information indicating a moving speed of the vehicle; and a thirddetection circuit configured to detect an abnormality in the first imageby comparing an optical flow for the first image estimated frominformation acquired by the information acquisition circuit and ashooting direction of the first camera with an optical flow for thefirst image calculated by the flow calculation circuit, and detect anabnormality in the second image by comparing an optical flow for thesecond image estimated from the information acquired by the informationacquisition circuit and a shooting direction of the second camera withan optical flow for the second image calculated by the flow calculationcircuit.
 11. An abnormality detection apparatus comprising: a memorystoring a program; a processor executing the program and configured to:extract a feature point and a feature value in a first image taken by afirst camera, and a feature point and a feature value in a second imagetaken by a second camera according to a predetermined extractionalgorithm, a shooting range of the second camera partially overlapping ashooting range of the first camera; calculate an optical flow for thefeature point of the first or second images extracted based on thefeature value of that feature point of the first or second images,respectively; detect an abnormality in the first image based on theoptical flow for the first image calculated and detect an abnormality inthe second image based on the optical flow for the second imagecalculated; and detect an abnormality in the first or second image basedon a result of a comparison between a first index value for the featurepoint of the first image in a first overlapped area and a second indexvalue for the feature point of the second image in a second overlappedarea, the first overlapped area being an area in the first image thatoverlaps the second image, the second overlapped area being an area inthe second image that overlaps the first image, wherein an extractionrange for the feature point and the feature value of the first andsecond images extracted is composed of a first partial area which is apredetermined partial area in the first image, a second partial areawhich is a predetermined partial area in the second image, and areasnear places in the first and second images predicted as destinations ofthe feature point of the first and second images.
 12. The abnormalitydetection apparatus according to claim 11, wherein the processor isfurther configured to: detect an abnormality in the first image based onthe an optical flow for the feature point in the first partial area anddetect an abnormality in the second image based on an optical flow forthe feature point in the second partial area, detect an abnormality byusing the feature point in a first overlapped extraction area defined inthe first overlapped area and the feature point in a second overlappedextraction area defined in the second overlapped area, wherein the firstoverlapped extraction area is an area in the first partial area thatcorresponds to the second partial area, and the second overlappedextraction area is an area in the second partial area that correspondsto the first partial area, and wherein the predetermined extractionalgorithm is commonly used when any abnormality is detected.
 13. Theabnormality detection apparatus according to claim 11, wherein the firstindex value is an optical flow for the feature point in the firstoverlapped extraction area, the second index value is an optical flowfor the feature point in the second overlapped extraction area, and thesecond abnormality detection unit compares an optical flow for thefeature point in a predetermined circular area designated in the firstoverlapped extraction area with an optical flow for the feature point ina predetermined circular area designated in the second overlappedextraction area.
 14. The abnormality detection apparatus according toclaim 13, wherein the processor is further configured to: use an opticalflow that is obtained by using a predetermined area in the firstoverlapped area as a tracking range for that optical flow, and anoptical flow that is obtained by using a predetermined area in thesecond overlapped area as a tracking range for that optical flow, anddetect an abnormality in the first and second images by using an opticalflow that is obtained by using a range wider than the tracking rangeused in the second abnormality detection unit as its tracking range. 15.The abnormality detection apparatus according to claim 11, furthercomprising of detecting an abnormality by comparing a first frequencydistribution and a second frequency distribution, the first frequencydistribution being a frequency distribution for numbers of samples foreach direction for the optical flow for the feature point in the firstoverlapped extraction area, the second frequency distribution being afrequency distribution for numbers of samples for each direction for theoptical flow for the feature point in the second overlapped extractionarea.
 16. The abnormality detection apparatus according to claim 11,further comprising to: make a final determination on presence/absence ofan abnormality in the first or second image based on detection resultsof the first and second abnormality detection units; determine whetheror not a detection result of the second abnormality detection unitshould be used for the final determination based on the number ofsamples in the first frequency distribution and the number of samples inthe second frequency distribution.
 17. The abnormality detectionapparatus according to claim 11, further comprising to: make a finaldetermination on presence/absence of an abnormality in the first orsecond image based on detection results of the first and secondabnormality detection units; and makes the final determination whilegiving a higher priority to the detection result of the secondabnormality detection unit than to the detection result of the firstabnormality detection unit.
 18. The abnormality detection apparatusaccording to claim 11, further comprising to detect an abnormality inthe first image based on whether or not temporal changes of a movingdirection and a moving amount of the optical flow for the first imageare within predetermined ranges, and detects an abnormality in thesecond image based on whether or not temporal changes of a movingdirection and a moving amount of the optical flow for the second imageare within predetermined ranges.
 19. A vehicle system including theabnormality detection apparatus of claim 11, further comprising: a firstcamera disposed in a vehicle, the first camera being configured to shoota surrounding of the vehicle; a second camera disposed in the vehicle sothat its shooting range partially overlaps a shooting range of the firstcamera, the second camera being configured to shoot a surrounding of thevehicle, and wherein the extraction of the feature point and the featurevalue in the first image is taken by the first camera, and the featurepoint and the feature value in a second image is taken by the secondcamera according to a predetermined extraction algorithm.
 20. Thevehicle system according to claim 19, further comprising: an informationacquisition device configured to acquire direction informationindicating a moving direction of the vehicle and speed informationindicating a moving speed of the vehicle; and a detection deviceconfigured to detect an abnormality in the first image by comparing anoptical flow for the first image estimated from information acquired bythe information acquisition device and a shooting direction of the firstcamera with an optical flow for the first image calculated, and detectan abnormality in the second image by comparing an optical flow for thesecond image estimated from the information acquired and a shootingdirection of the second camera with an optical flow for the second imagecalculated.